Keep idle SSH tunnel sessions alive on dedicated clusters - #6382

Stacked on #6358: the base branch is that PR's branch, so this diff is only the change on top of it. GitHub retargets to main when #6358 merges.

Changes

#6358 fixes idle-session drops on serverless and does nothing for a dedicated cluster. Its websocket ping is a control frame: the driver proxy terminates it itself, so it never becomes payload on the leg past it, and that leg is reaped after ~8m20s of carrying nothing. Zero pongs came back in any run on either compute type — the frame does not traverse. Serverless sits behind a different reaper that the ping does reset, which is why it fixes serverless and only serverless.

The fix is to generate traffic the tunnel forwards end to end. An SSH keepalive reply is a real SSH packet, so the proxy loops carry it as a websocket data frame across every hop — which is why setting ServerAliveInterval by hand is a total workaround. The CLI now asks for it itself, from both ends:

• ServerAliveInterval 30 in buildSSHArgs (the connect path) and GenerateHostConfig (the setup and --ide paths). Needs nothing from the compute, so it also fixes a cluster already running a server binary from an older CLI.
• ClientAliveInterval 30 in the sshd config the tunnel's server writes. Covers clients the CLI never configures — a hand-written ProxyCommand block, or an IDE supplying its own ssh options.

Either half alone is sufficient (measured below). Both are in because they close different gaps and neither subsumes the other. The websocket ping stays: it is what fixes serverless, and it keeps the client↔control-plane hop warm regardless of the user's ssh config.

One behaviour change, deliberately. These options bring in ServerAliveCountMax / ClientAliveCountMax (OpenSSH default 3), so a tunnel that stops responding is now torn down after ~90s with ssh's own "server not responding" message instead of hanging. That is the trade we want — today a reaped idle session is a silent black hole, which is the original report — and 90s is well clear of the longest legitimate pause on a healthy tunnel, the up to 30s a handover can hold the sending loop (proxyHandoverInitTimeout).

sshdConfigContent is prepareSSHDConfig's config string moved verbatim into a function plus the one new line, so the content is testable without mocking three secret reads. #6358's changelog fragment is updated rather than joined by a second, contradicting one.

An out-of-band keepalive frame of our own was rejected: runReceivingLoop writes every binary message straight into the SSH byte stream, so it needs a framing change on both ends and only helps once the uploaded server binary is new enough.

Resolves DECO-28186 for dedicated clusters; #6358 resolves it for serverless.

Why

Dedicated compute is what the README requires for remote development in an IDE, so the compute type most affected was the one #6358 left broken. Shipping only the websocket ping would close the ticket with the reported failure still in place for those users, and add a log line that makes an idle session look healthy while it is dying.

Tests

Three unit tests, one per site the directive has to appear in, each also bounding the interval against the reap window and against the handover pause. All six mutants killed: dropping the option from each of the three sites, setting either constant to 0, and pushing the interval past the reap window.

./task test-exp-ssh, ./task lint (0 issues, all three modules), ./task fmt, and the whitespace, deadcode and changelog checks are all clean; the four touched packages are green under -race.

End-to-end (dogfood, dedicated clusters)

One dedicated single-node cluster per build — two tunnels on one cluster collide on port 7772 and take each other down. DBR 17.3 LTS, m5d.xlarge, SINGLE_USER; distinct pinned version per build, since uploadReleases skips the upload when the versioned workspace path already exists. Every session ran on a real PTY, was left completely untyped for the whole window, then sent exactly one command. Idle is measured remote-clock to remote-clock.

• dc-base reproduces the bug on Keep idle SSH tunnel sessions alive with a websocket ping #6358's own head, as reported: alive-looking for the whole window, killed by the first keystroke. The server lost its half at 18:04:46 — 8m20s after the session went quiet — with websocket: close 1006 (abnormal closure): unexpected EOF. The client had sent 30 pings with zero failures by then; its next two failed, after the server was already gone. Ended close 4000: Handler crashed: ...ClosedStreamException, exit status 255.
• dc-sshd-only isolates the server half: same build, but ssh invoked by hand with -o ServerAliveInterval=0, so sshd drives the only SSH-level keepalive. A command-line -o beats any config file in OpenSSH, and ClientAliveInterval 30 was read back off the cluster's generated sshd_config. The complementary control for the client half — old server binary, client-side option only — passed earlier on the same setup.
• dc-fix-long is past two reap windows, so the PASS is a real fix rather than a delayed drop.

Full measurements and log excerpts are on DECO-28186. The test clusters have been terminated.

This PR and its description were written by Isaac.